Complexity Preserving Functions

It’s widely recognized that compression is useful and even necessary for inductive learning, where a short description will capture the ‘regularities’. We introduce complexitypreserving functions that preserve these regularities of the concept. They are based on the universal information distance [Bennett et al. ‘97] and define for an instance a set of elements sharing the same complexity type. This corresponds to the two-part code [Rissanen ‘89] of the MDL principle, when it is interpreted as the first term describing the set and the second term the element in the set [Rissanen 99]. We investigate its importance in inductive learning. Introduction The Kolmogorov complexity of a string x K(x) is the length of a shortest program to compute x on a universal computer. It represents the minimal amount of information required to generate x by an effective process [Kolmogorov 65]. Bennett et al. studied the definition of a universal cognitive distance [Bennett et al. ‘97], based on the observation that “the psychological similarity between two objects x and y is the complexity of the simplest transformation between them” [see also Chater 2003]. They start by defining the shortest program p to compute y from x: KF= min{l(p) : F(p, x) = y} Where F(p,x) is a partial recursive function, that can be computed on a Turing machine and l(p) denotes the (binary) length of vector p. To attain a symmetric definition, Benett et al. investigate reversible computations, where the overlap between the information to convert x into y and y into x is maximized. We try to define functions, by using the notion of information distance, that not only preserve complexity, but also capture the ‘complexity type’ or ‘regularity’ of an object. K-preserving functions A Kolmogorov complexity-preserving function gx is a one-to-one reversible function defined from (X, R) -> X: x’ = gx(x, k). Except for a limited number of special k values, where the K-complexity is reduced in x’, what we call the reduction values, the following constraints apply for all k: 1. x = gx(x’, -k). 2. K(gx,k|x) < K(gx, k), x contains information to perform gx. 3. The program p to compute gx is both a minimal program, for computing x’ from x and vice versa. This last condition means that there is no extra information needed for either computation, there is maximum overlap of information and the complexity is similar in x and x’. Hence, the difference between the complexities of x and x’, K(x) and K(x’), depends only on k, not on gx. The differences between the complexities is bounded by |K(x) K(x’)| ≤ l(k). There is a fourth constraint on the implementation of the k-values: 4. gx (x, a.k + b.l) = a.gx(b.gx(x, l), k) Examples of K-preserving functions are translations, rotations, enlargements, repetitions, polynomials, permutations,... On a circle, translations in each direction and resizing preserve the complexity. An ellipse can additionally be stretched along both its axes, however, for a special reduction value k, the ellipse becomes a circle, that is of lower complexity. Set with the same regularities By applying all existing gx functions on x by varying the k values, we become a set Cx of elements with the same regularity. Each element of this set got the same K-preserving functions and will generate the same set (following from constraint 4). This means that they all share the same regularity. There are dependencies among the Kpreserving functions. If for a certain x: gx,1(x, k1) = gx,3(gx,2(x, k2), k3), (with all k nonreduction values), gx,1 can be composed out of gx,2 and gx,3 for all elements of Cx. After elimination of the dependent gx,i, a set of independent K-preserving functions Cg{gx,i} remains, which represents the degrees of freedom of the regularity. The set Cx got Πki elements and each element in the set is represented by its (k1, k2, ...), after having chosen a reference element (with all k’s set to